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ABSTRACT 

The  key  component  in  developing  an  effective  virtual  infantry  training  simulator  is  the  user  interface.  Our 
goal  is  to  develop  interfaces  that  give  users  close  to  the  same  ability  to  move  and  coordinate  actions  as  they 
have  in  the  real  world.  We  have  developed  two  interfaces.  Gaiter  is  a  highly  realistic  body-driven  interface 
in  which  the  user  walks  in  place  to  walk  through  the  virtual  world.  With  Gaiter,  the  user  can  naturally 
intermix  a  range  of  natural  and  gestural  actions.  An  experiment  comparing  Gaiter  with  less  realistic 
interfaces  shows  that  a  control  technique  that  mimics  a  user’s  natural  actions,  while  beneficial,  does  not 
immediately  provide  all  the  capabilities  of  natural  motion.  These  results,  along  with  a  Marine  Corps  interest 
in  lower  cost,  more  deployable  systems,  have  led  us  to  develop  a  new  virtual  locomotion  control.  Pointman  is 
a  device-driven  interface  that  uses  a  conventional  dual  joystick  gamepad.  Unlike  the  control  mappings  of  a 
conventional  gamepad,  Pointman  allows  the  user  to  specify  direction  of  movement  independently  from  the 
heading  of  the  upper  body,  allowing  the  user  to  execute  realistic  tactical  infantry  movements  such  as  pie-ing 
the  corner.  Pointman  can  also  be  used  for  teleoperation  of  remotely  piloted  vehicles,  providing  added 
separation  over  the  vehicle ’s  motion  and  view.  Both  interfaces  derive  from  an  analysis  of  action  and  effect 
that  highlights  the  importance  of  providing  open  loop  control. 

1.0  INTRODUCTION 

The  user  interface  is  the  key  component  in  developing  an  effective  virtual  infantry  training  simulator.  Our 
goal  is  to  design  and  implement  interfaces  that  give  trainees  close  to  the  same  ability  to  move  and  coordinate 
actions  as  they  have  in  the  real  world.  We  have  developed  Gaiter ,  a  full-body  immersive  interface  in  which 
the  user  walks  in  place  to  walk  through  the  virtual  environment  (VE).  We  have  also  recently  implemented  a 
new  control,  called  Pointman,  using  a  conventional  gamepad.  Pointman  has  a  novel  control  mapping  that 
allows  users  to  turn  their  body  independently  of  their  direction  of  motion.  View  control  is  offloaded  to  inertial 
sensors.  Pointman  makes  it  easy  for  the  user  to  execute  tactical  movements  such  as  pie-ing  a  corner,  which  is 
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difficult  to  do  using  conventional  controls.  Pointman  allows  the  user  to  perform  training  exercises  following 
prescribed  military  tactics,  techniques,  and  procedures  (TTPs)  using  a  compact,  low  cost  interface.  Pointman 
can  also  be  used  for  teleoperation  of  remotely  piloted  ground  vehicles  to  increase  the  pilot’s  freedom  and 
control  over  the  vehicle’s  motion  and  allow  them  to  execute  more  effective  tactical  maneuvers. 

To  be  effective,  the  user  interface  should  allow  the  user  to  control  his  or  her  avatar,  an  articulated 
representation  the  user’s  body  in  the  virtual  world,  in  as  natural  a  way  as  possible.  By  that  we  mean  that  users 
should  be  able  to  move  their  avatars  in  the  virtual  world  as  they  would  move  their  physical  bodies  in  a 
corresponding  real  world  environment.  Our  two  interfaces  represent  two  different  levels  of  realistic 
interfaces.  Gaiter  is  an  example  of  body-driven  interaction  in  which  the  user  performs  actions  by  moving  his 
or  her  own  body  to  either  execute  or  mimic  the  performance  of  corresponding  real  world  actions.  Pointman  is 
an  example  of  device-driven  interaction  in  which  the  user  specifies  the  motion  of  his  or  her  avatar  through 
manipulating  an  input  device.  Device-driven  interfaces  are  by  far  the  more  widely  adopted  approach,  as  can 
be  seen  in  the  use  of  joysticks,  gamepads,  and  mouse-keyboard  interfaces  to  direct  one's  avatar  through  a  wide 
variety  of  military  simulations  and  first-person  computer  games.  There  are  many  different  ways  of  controlling 
one's  avatar  via  device-driven  interaction  and  some  forms  of  control  produce  more  natural  motions  than 
others.  Within  the  specific  domain  of  infantry  simulation,  we  will  show  that  controls  designed  to  support 
tactical  infantry  movement  produce  better  results  than  conventional  game  controls. 

The  benefits  of  virtual  simulators  for  individual  combatants  are  substantial.  Accurate  representation  of  the 
behavior  of  infantrymen  remains  a  missing  link  in  military  simulations.  The  potential  is  to  offer  training  and 
mission  rehearsal  anytime  and  anywhere  to  maintain  sustainment  levels  and  reduce  skill  decay.  Moreover, 
virtual  simulators  are  the  only  way  to  directly  train  scenarios  that  would  be  too  costly  or  too  dangerous  to 
practice  in  the  real  world.  Scenarios  can  familiarize  trainees  with  operating  in  dangerous  environments  to 
help  acclimate  them  to  the  stress  of  close  confrontation.  To  date,  virtual  simulators  have  not  been  widely 
adopted  because  until  recently  the  equipment  has  been  too  costly  and  system  latency  higher  than  desired.  The 
opportunity  is  now  here  to  create  effective  training  simulators. 


2.0  DESIGN  PRINCIPLES  FOR  REALISTIC  INTERFACES 

The  development  of  all  realistic  motion  interfaces  requires  an  analysis  of  the  perceptual/action  issues  in  the 
task  domain  to  determine  how  actions  are  coordinated  to  produce  effects.  In  the  real  world,  a  person  performs 
actions  by  activating  muscles  to  selectively  apply  force  and  move  parts  of  the  body.  These  actions  produce 
effects,  which  alter  the  person’s  relationship  with  the  environment.  People  balance  while  successively 
swinging  their  legs  apart  and  together  to  move  their  body  forward.  In  an  analogous  manner,  users  perform 
control  actions  that  produce  control  effects  in  the  virtual  world.  The  goal  is  to  develop  interfaces  that  allow 
users  to  naturally  coordinate  motion  and  to  execute  a  variety  of  tactical  infantry  movements  in  a  realistic 
manner. 

To  aid  in  describing  a  realistic  interface,  we  have  created  a  list  of  properties  derived  from  an  analysis  of  how 
people  perform  real  world  actions  to  achieve  effects  [17].  The  list  contains  the  salient  properties  of  both  real 
world  and  control  actions  and  effects.  The  list  applies  to  both  highly  realistic  body-driven  interfaces  as  well 
as  more  abstract  device-driven  ones;  the  difference  is  in  the  level  of  fidelity  of  each  property.  The  six 
properties  help  describe  the  nature  of  the  match  between  real  world  actions  and  effects,  and  control  actions 
and  effects. 

The  first  three  properties  only  describe  actions:  (1)  body  segments  (which  body  segments  are  involved  in 
producing  the  action(s)),  (2)  effort  (how  much  effort  is  required),  (3)  coordination  (what  is  the  person’s  ability 
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to  perform  the  actions  in  concert).  The  next  two  properties  apply  to  both  actions  and  effects:  (4)  degrees-of- 
freedom  and  range  of  motion  (what  degrees-of-freedom  and  range  of  motion  are  used  in  executing  the  action 
and  displayed  as  the  effect),  (5)  rate  and  accuracy  (what  is  the  rate  and  accuracy  with  which  a  motion  can  be 
performed  and  displayed).  The  last  property  concerns  how  actions  and  effects  work  together:  (6)  open  or 
closed  loop  control  (can  the  action  be  achieved  without  sensory  feedback  under  an  open  loop  control  or  is 
feedback  required).  All  six  properties  are  important  but  providing  an  open  loop  control  where  appropriate  can 
be  critical  for  realism.  For  example,  people  can  maintain  their  sense  of  direction  over  short-range  turns  and 
translations  in  the  dark  [11],  which  is  used  in  night  operations.  However,  it  can  be  a  demanding  property  to 
achieve  in  an  interface. 

For  body-driven  interfaces,  whenever  possible,  interaction  should  be  one-for-one  to  provide  the  best  match, 
such  as  when  turning  the  body  or  carrying  and  aiming  a  rifle.  However,  not  all  actions  can  be  one-for-one 
because  the  user  inhabits  a  virtual  world  containing  virtual  objects  that  is  larger  than  the  immediate 
surroundings.  In  that  case,  we  analyze  the  characteristics  of  the  real  world  action  in  terms  of  the  six  properties 
and  select  a  gestural  control  action  that  closely  mimics  the  natural  action.  For  the  gestural  control  action  for 
locomotion,  a  scaled  mapping  is  used  to  move  the  avatar  forward  at  an  appropriate  walking  or  running  rate. 
For  one-for-one  control  actions,  actions  made  by  the  user  are  directly  mapped  into  the  motion  of  the  avatar. 

For  device-driven  interfaces,  the  properties  describing  actions  and  effects  still  apply  but  in  a  slightly  different 
way.  The  properties  for  action  now  apply  to  the  control  actions  that  have  correspondence  with  real  world 
actions.  These  control  actions  should  afford  natural  movement  and  tap  into  a  user’s  innate  skills  and  abilities. 
They  should  express  sufficient  information  to  produce  realistic  effects  in  the  virtual  world.  The  properties  for 
effect  apply  as  they  did  for  body-driven  interfaces.  Users  must  be  able  to  control  how  their  avatars  look, 
move,  and  shoot  with  the  same  capabilities  and  limitations  as  people  have  in  the  real  world.  If  motion  is  not 
expressed  realistically,  users  develop  an  unrealistic  expectation  of  what  is  possible. 


3.0  GAITER 

The  Gaiter  body-driven  interface  [16]  allows  users  to  walk  and  run  in  place  to  control  their  movement  through 
the  virtual  world.  Gaiter  was  originally  designed  to  provide  directional,  leg-based  control  over  virtual 
locomotion.  The  direction  of  motion  of  the  user's  avatar  is  controlled  by  the  direction  and  extent  of  movement 
of  the  user’s  legs.  In  this  way,  Gaiter  operates  in  a  body-centric  coordinate  frame  of  reference.  Movements  of 
other  parts  of  the  avatar  match  the  user's  actions  one-for-one.  This  allows  users  to  naturally  intermix  a  wide 
range  of  upper  and  lower  body  movements  to  execute  a  wide  range  of  actions.  Later  the  Gaiter  interface  was 
extended  to  support  full-body  tracking  and  aiming  with  a  rifle  prop  for  use  as  an  individual  combatant  VE 
simulator  to  train  Marines  in  CQB  for  urban  warfare. 

In  the  current  version  of  the  system,  the  user  wears  an  NVISOR  SX  model  head-mounted  display  (HMD) 
from  NVIS,  Inc.  to  provide  a  360°  fleld-of-regard.  The  image  array  is  1280  by  1024  per  eye  with  a  48° 
horizontal  by  36°  vertical  fleld-of-view.  It  provides  a  stereoscopic  image.  The  user  is  outfitted  with  a 
standard  flak  jacket  and  carries  an  instrumented  rifle  prop,  a  full-weight  M4  replica  Airsoft  AEG  model  that 
transmits  a  signal  when  the  trigger  is  pulled.  The  user  is  attached  to  a  centering  harness  developed  by  Dr. 
Roger  Kaufman  at  George  Washington  University,  which  is  made  of  steel  tubing  that  acts  as  a  spring  and 
allows  the  user  to  move  no  more  than  a  foot  off  center.  The  harness  turns  freely  and  permits  the  immersed 
user  to  turn,  walk,  and  run  in  place  while  staying  in  the  center  of  the  tracked  area.  The  user’s  body  is  tracked 
using  the  Vicon  612  model  passive  optical  tracking  system  with  10  cameras.  Tracking  markers  are  placed  on 
rigid  objects  (HMD,  flak  jacket,  rifle,  harness  waist  belt,  and  arm  and  shin  guards)  to  track  individual  body 
segments. 


RTO-MP-HFM-1 36 


23-3 


Immersive  Simulation  to  Train  Urban  Infantry  Combat 


ORGANIZATION 


The  Gaiter  interface  includes  one-to-one  motion  mapping  for  looking,  turning,  and  rifle  handling.  For 
shooting,  the  user  sees  a  virtual  representation  of  a  rifle  co-aligned  with  the  physical  rifle  and  is  able  to  look 
through  the  rendered  sights  to  aim  at  the  target.  The  location  of  the  hit  is  determined  from  the  pose  of  the  rifle 
muzzle  in  the  virtual  world.  In  this  way,  the  correct  perceptual/action  skills  are  used  to  locate  the  target,  put 
the  rifle  on  the  target,  and  keep  the  rifle  on  the  target  while  the  shot  is  fired.  Since  the  HMD  shows  the 
imagery  on  a  single  focal  plane,  the  skill  of  focusing  the  eyes  on  the  front  sight  is  not  trained.  We  blur  the 
image  of  the  rear  sight  to  add  realism  but  the  degree  of  blurring  is  fixed  and  does  not  depend  on  user’s  depth 
of  focus. 

The  Gaiter  locomotion  technique  is  a  gestural  technique  that  follows  the  design  guidelines  for  body-driven 
interfaces.  Walking  and  running  in  place  have  similarities  with  natural  walking  and  running.  Both  allow  the 
legs  to  be  moved  in  any  direction;  both  are  compatible  with  other  actions  and  permit  a  natural  degree  of 
coordinated  action;  and  both  operate  in  a  natural  body-centric  coordinate  system  that  allows  reflexive 
orientation.  Since  turning  is  performed  by  physically  stepping  to  turn,  Gaiter  provides  correct  vestibular  and 
proprioceptive  cues  for  turning.  Slater,  Usoh,  and  Steed  [13]  have  demonstrated  that  the  proprioception 
provided  by  a  walking  in  place  technique  tends  to  increase  subjective  presence,  which  is  a  sense  of  actually 
being  in  the  virtual  space,  when  there  is  an  association  with  an  avatar.  Through  informal  use,  we  have  also 
found  that  users  have  a  good  sense  of  being  surrounded  by  the  virtual  world  when  using  Gaiter. 

Gaiter  has  been  shown  to  work  well  in  a  usability  study  testing  path  following  tasks  [14].  The  results  show 
that  there  was  a  consistent  accuracy  (an  RMS  error  of  about  six  inches)  and  speed  (of  around  five  feet  per 
second)  associated  with  walking  forward  down  a  three-foot  path.  Also,  an  assessment  of  simulator  sickness 
using  the  questionnaire  developed  by  Kennedy,  Lane,  Berbaum,  and  Lilienthal  [5]  showed  that  simulator 
sickness  was  not  a  problem.  Subjects  did  sweat  but  sweating  was  associated  with  wearing  a  flak  jacket, 
HMD,  and  stepping  in  place.  In  general,  Gaiter  allows  the  user  a  fair  level  of  accuracy  of  control  over 
locomotion  and  permits  open  loop  control  over  turning.  Users  can  coordinate  actions  in  tasks  such  as  looking 
into  vistas  opening  up  as  they  move  around  corners,  holding  the  rifle  at  the  ready  while  moving  and  searching 
for  targets,  and  transitioning  into  a  good  firing  position. 

The  Gaiter  interface  is  included  in  a  series  of  studies  looking  at  the  effectiveness  of  interfaces  for  training. 
We  are  conducting  these  studies  in  collaboration  with  the  Naval  Research  Laboratory’s  Warfighter  Human- 
System  Integration  Laboratory  (WHSIL).  These  studies  are  part  of  the  VIRTE  (Virtual  Technologies  and 
Environments)  program,  which  is  funded  by  the  Office  of  Naval  Research.  Preliminary  results  from  the  first 
experiment  are  below  in  section  3.1. 

3.1  Comparison  of  Interfaces  to  Support  Path  Integration 

The  first  experiment  looked  at  how  well  subjects  performed  path  integration  tasks  using  one  of  three 
interfaces.  Path  integration  is  an  aspect  of  navigation  in  which  a  person  uses  velocity  and  acceleration  signals 
from  proprioceptive  and  kinesthetic  inputs  to  monitor  one’s  position  along  a  pathway. 

The  three  interfaces  tested  were  Gaiter,  Rifle-Mounted  Joystick,  and  Desktop  Joystick.  Each  interface  offered 
a  different  functionality.  Gaiter  allowed  subjects  to  control  the  rate  and  direction  of  movement  with  their  legs 
and  to  turn  naturally.  With  the  Rifle-Mounted  Joystick,  the  rate  and  direction  of  translation  was  controlled 
with  a  thumb-joystick  mounted  on  the  handguard  of  a  rifle  prop.  (The  thumb-joystick  was  mounted  on  the  left 
side  of  the  handguard  and  only  right-handed  subjects  could  participate).  Subjects  controlled  translation  by 
pushing  the  thumb-joystick  in  the  desired  direction  relative  to  where  the  rifle  was  pointed.  The  subject  pushed 
the  thumb-joystick  forward  to  go  forward,  up  to  go  right,  back  to  go  backwards,  and  down  to  go  left.  Turning 
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was  achieved  by  turning  the  body.  Subjects  using  the  Desktop  Joystick  controlled  rotation,  rate,  and  direction 
of  movement  with  the  stick.  Twisting  the  stick  rotated  the  view.  Subjects  translated  their  avatars  by  pushing 
the  stick  in  the  desired  direction.  The  direction  of  translation  was  specified  with  respect  to  the  view  direction. 
A  hat  control  tilted  the  image  up  and  down  and  a  reset  button  re-centered  the  view.  All  conditions  included  a 
trigger  pull  to  mark  events.  The  display  was  held  constant:  all  subjects  wore  the  NVISOR  SX  model  HMD. 
The  image  moved  with  head  motion  in  the  Gaiter  and  Rifle-Mounted  Joystick  conditions;  the  image  only 
changed  under  joystick  control  in  the  Desktop  Joystick  condition. 

The  experiment  consisted  of  three  tasks.  The  maze  task  was  based  on  pointing  to  origin,  which  has  been 
shown  to  assess  path  integration  ability  [8].  We  constructed  a  series  of  passageways  with  two  halls  and  an 
intervening  turn,  and  asked  the  subjects  to  point  back  to  the  origin  at  the  end.  We  adapted  the  design  from  a 
study  by  Klatzky,  Loomis,  Beall,  Chance,  and  Golledge  [6].  Blind  walking  is  a  version  of  open  loop 
responding,  which  has  also  been  shown  to  directly  assess  path  integration  [8].  Researchers  have  demonstrated 
that,  in  the  real  world,  people  have  the  ability  to  look  at  an  object  with  the  lights  on,  turn  off  the  lights,  and 
walk  to  that  location,  on  average,  with  little  systematic  error  out  to  about  15  m  [such  as  in  3,7,  10,  11,  12]. 
The  third  task,  blind  rotations ,  compared  the  ability  of  subjects  to  rotate  a  given  angle  without  vision. 
Rotation  is  an  important  component  of  locomotion  and  associated  with  path  integration.  It  was  included 
because  it  was  easy  to  isolate  and  provided  additional  insights.  Because  the  image  was  viewed  through  the 
same  HMD,  the  differences  in  indicated  direction  or  distance  among  the  locomotion  interfaces  reflected  how 
well  the  subjects  used  each  interface  and  were  able  to  calibrate  to  it  during  practice.  Ideally,  the  subject  would 
acquire  a  sense  of  the  space  while  moving  through  the  practice  environments  and  perform  navigation  tasks 
close  to  how  they  would  in  the  real  world. 

The  study  design  was  between  subjects  with  eight  subjects  per  group.  All  subjects  were  employed  at  the 
Naval  Research  Laboratory.  All  had  good  vision  and  all  but  three  were  male,  reflecting  NRL’s  workforce. 
Their  ages  ranged  between  26  and  35.  Before  the  series  of  tasks,  the  experimenter  demonstrated  the  interface 
to  the  subject  and  the  subject  had  an  opportunity  to  try  it  in  a  practice  environment.  The  order  of  the  tasks  was 
the  maze  task,  blind  walking,  and  blind  rotations. 

The  maze  task  was  the  only  task  performed  with  visual  feedback  throughout.  The  subject  started  the  task  in  a 
3  m  square  virtual  room.  The  first  hall  was  9  m  long;  the  second  hall  was  5  m.  The  turn  occurred  in  a  round 
room  3  m  in  diameter.  The  room  where  the  response  was  given  was  also  3  m  square.  The  angles  of  the  turn 
were  45°,  90°,  or  135°.  The  connecting  doors  closed  silently  behind  the  subject  as  they  entered  a  room  and 
blended  into  the  wall  once  closed.  In  the  last  room,  crosshairs  appeared  in  the  subject’s  field-of-view  upon 
reaching  the  center  of  the  room,  and  the  subject  rotated  his  or  her  view  to  point  back  at  the  start  location  and 
pulled  the  trigger.  The  subject  was  then  transported  to  the  start  location  to  begin  a  new  trial.  There  were  six 
trials  with  each  angle  repeated  twice.  The  results  of  the  three  angle  repeated  measures  test  showed  that 
interface  was  significant  (F(2,  90)  =  4.47,  p  =  .024).  Scheffe  post  hoc  comparisons  indicated  that  the  Desktop 
Joystick  interface  had  significantly  less  angular  error  than  the  Rifle-Mounted  Joystick  (p  =  .046). 
Performance  with  Gaiter  was  mixed  but  there  was  no  device-by-angle  interaction.  Gaiter  performed  more  like 
the  Rifle-Mounted  Joystick  for  the  45°  turn  but  more  like  the  Desktop  Joystick  for  90°  and  135°. 

We  based  the  blind  walking  task  on  a  design  by  Philbeck,  Loomis,  and  Beall  [10].  We  included  both  direct 
and  indirect  walking  as  they  did;  however,  due  to  technical  problems,  the  indirect  walking  data  were  not 
usable  although  the  task  still  served  as  a  distractor  so  subjects  could  not  pre-program  their  response  for  direct 
walking.  Only  direct  walking  will  be  described.  The  task  took  place  in  a  large  virtual  warehouse  that  had 
three  1.25  m  wide  paths  marked  on  the  floor  with  a  pair  of  colored  strips.  The  subject  faced  down  the  center 
path  and  an  object,  a  5’  1 1”  steel  plate  target  of  a  combatant,  was  placed  on  the  path  at  one  of  three  locations, 
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3.33  m,  6.66  m,  or  10  m,  from  the  start  location.  Each  distance  was  repeated  three  times.  The  subject  viewed 
the  object  for  5  sec  until  the  image  faded.  Blind  walking  experiments  in  the  real  world  ask  subjects  to  close 
their  eyes  but  that  was  impossible  to  confirm  with  an  HMD  so  we  faded  the  image.  Only  the  pair  of  stips 
bounding  the  path  was  visible  to  the  subject  to  help  the  subject  maintain  lateral  stability.  The  subject  was 
instructed  to  walk  forward  to  where  the  object  had  been  located,  stop  at  that  location,  and  press  the  trigger. 
An  analysis  of  distance  travelled  along  the  path  showed  that  interface  was  significant  (F(2,  100  =  3.53),  p  = 
.047).  Bonferroni  post  hoc  comparisons  indicated  that  the  Rifle-Mounted  Joystick  was  more  accurate  than 
Gaiter  (p  =  .05).  However,  the  analysis  does  not  tell  the  whole  story.  An  interesting  issue  was  raised  in  the 
question  session  after  the  experiment.  A  few  subjects  said  their  strategy  using  Gaiter  was  to  double  the 
number  of  in-place  steps  they  took  in  VE  to  estimate  how  many  actual  steps  it  would  take  in  the  real  world. 
The  results  agree  with  this  observation  because  the  subjects  travelled  about  twice  as  far  as  they  should  have. 
When  distance  travelled  was  divided  by  2,  interface  was  no  longer  significant.  What  appears  to  have 
happened  is  that  the  highly  analytical  NRL  subjects  interpreted  their  in-place  steps  as  half  steps  rather  than 
whole  steps  when  confronted  with  the  blind  walking  task. 

In  blind  rotations ,  subjects  rotated  45°,  90°,  or  180°  left  or  right  with  the  image  in  the  HMD  faded.  Each  angle 
and  direction  was  repeated  twice.  The  results  show  a  significant  difference  for  interface  (F(2,  146)  =  5.72,  p  = 
.01).  Scheffe  post  hoc  comparisons  showed  that  the  Desktop  Joystick  was  significantly  worse  than  either 
Gaiter  (.02)  or  the  Rifle-Mounted  Joystick  (p  =  .036).  The  results  confirm  the  advantage  of  an  interface  that 
provides  proprioceptive  and  kinesthetic  input  for  turning. 

The  overall  conclusion  suggests  that  when  visual  cues  are  present,  subjects  perform  well  with  the  Desktop 
Joystick.  When  vision  is  not  available,  proprioceptive  and  kinesthetic  input  helps,  although  the  results  are 
mixed.  For  blind  rotations,  actual  turning  used  in  Gaiter  and  the  Rifle-Mounted  Joystick  improved 
performance  over  the  rate  based  Desktop  Joystick,  which  is  consistent  with  previous  research  [2].  The  results 
for  direct  blind  walking  along  a  path  were  confounded.  It  was  expected  that  interfaces  with  proprioceptive 
and  kinesthetic  input  from  walking,  like  Gaiter,  would  result  in  better  performance.  But  the  Gaiter  results 
contained  a  consistent  miscalibration  in  which  subjects  interpreted  in-place  steps  as  half  steps  and  travelled 
twice  as  far  as  necessary.  After  adjusting  the  Gaiter  data,  the  results  showed  no  significant  difference  between 
in-place  walking  and  the  two  joystick  conditions.  Further  study  is  needed  to  see  if  users  can  accept  a  whole 
step  interpretation  of  their  motion  achieved  through  walking  in  place. 


4.0  POINTMAN 

Pointman  is  a  new  virtual  locomotion  control  based  on  a  dual  joystick  gamepad  with  added  sensors.  Unlike 
conventional  gamepad  control  mappings  for  first-person  shooter  games,  the  user  can  control  the  locomotion  of 
his  or  her  avatar  in  a  way  that  offers  more  effective  control  over  a  wide  range  of  tactical  motions.  Pointman 
allows  users  to  scan  their  view  and  rifle  to  cover  danger  areas  without  disrupting  their  motion  along  a  path. 
Few  controls  provide  this  ability,  yet  the  majority  of  tactical  infantry  movements  depend  upon  it.  It  permits 
individuals  to  control  realistic  tactical  movements  in  virtual  simulations.  Since  many  of  these  tactical 
movements  relate  to  how  an  individual  operates  as  part  of  a  larger  team,  this  new  locomotion  control  permits 
realistic  small  unit  team  tactics  to  be  simulated. 

This  work  was  motivated  by  an  examination  of  the  small  unit  infantry  tactics  employed  in  urban  combat  in 
general  and  building  clearing  operations  in  particular.  We  observed  how  experts  perform  tactical  movements 
in  the  real  world  and  using  Gaiter  in  simulation.  We  compared  this  with  the  way  people  control  the 
movement  of  their  avatars  with  conventional  gamepad  and  mouse-keyboard  interfaces  used  in  first-person 
shooter  games  and  observed  different  movement  strategies  being  used. 
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Tactical  infantry  movement  involves  keeping  the  rifle  directed  where  you  are  looking  while  scanning  for 
threats.  ’’The  purpose  of  any  tactical  maneuver  is  to  allow  your  muzzle  to  cover  the  potential  danger  areas  as 
you  encounter  them.  Observe  the  three-eye  principle.  This  means  that  your  weapon  must  be  oriented  toward 
whatever  it  is  your  eyes  are  looking  at.  Wherever  your  eyes  go,  your  weapon  must  also  go.  Keep  the  weapon 
in  a  ready  position  or  'hunting’  attitude  so  that  it  does  not  obstruct  your  vision  while  you  search"  [15,  p.22]. 
Figure  1  illustrates  two  tactical  movements.  Scanning  is  used  to  cover  and  clear  danger  areas,  especially  areas 
exposed  as  one  moves  around  comers.  In  terms  of  perceiving  the  environment  during  locomotion,  vistas  open 
at  the  occluding  edge  of  a  comer  as  a  person  advances  [4,  p.  199].  In  terms  of  infantry  tactics,  to  ’pie'  a  comer 
one  directs  one's  attention  just  past  the  corner's  edge  to  look  into  the  area  incrementally  disclosed  by  moving 
along  a  hallway,  leading  to  the  corner. 


Figure  1:  Left:  Turning  the  Heading  towards  a  Corner  while  Maintaining  a  Straight  Course. 
Turning  the  upper  body  to  face  the  corner  is  a  tactically  correct  way 
to  clear  a  corner.  Right:  Pie-ing  past  an  open  doorway. 


Scanning  is  also  used  to  direct  one's  view  and  rifle  from  side-to-side  to  cover  one's  sector  of  responsibility 
when  moving  as  part  of  a  team  through  an  open  area,  such  as  a  street  or  courtyard.  Different  members  of  a 
patrol  team  are  responsible  for  providing  front,  flank,  or  rear  security.  Once  something  of  interest  is  found, 
targeted  scanning  motion  is  used  to  remain  focused  on  it. 

The  tactics  executed  with  conventional  gamepads  place  a  greater  emphasis  on  moving  sideways  and  spiraling 
towards  or  away  from  points  of  interest,  due  primarily  to  the  limitations  of  the  control.  Once  that  difference 
was  clear,  we  saw  the  opportunity  to  develop  a  small  footprint,  low  cost  virtual  locomotion  control  using 
conventional  input  devices  that  could  better  approximate  the  kinds  of  movements  involved  in  tactical  infantry 
maneuvers.  The  resulting  control  can  benefit  any  application  where  it  is  useful  to  freely  look  around  while 
continuing  to  move. 

4.1  Framework  for  Describing  Locomotion 

We  developed  a  framework  for  classifying  real  world  locomotion  that  we  used  to  specify  Pointman  and 
describe  the  performance  of  conventional  gamepads  for  first-person  shooter  games.  Although  control  over 
turning  the  view  from  side-to-side  and  pitching  the  view  up  and  down  is  vital,  it  will  be  treated  separately 
from  this  framework. 

An  important  goal  of  virtual  locomotion  is  to  retain  the  ability  to  coordinate  the  motion  of  the  avatar  when  we 
substitute  control  using  an  input  device  for  natural  ways  of  moving.  We  want  motion  in  the  virtual  world  to 
closely  resemble  the  motions  a  person  would  make  in  the  real  world.  In  the  case  of  tactical  infantry 
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movements,  the  specific  kinds  of  motion  are  spelled  out  in  the  Marine  Corps’  field  manual  [9].  In  general, 
when  executing  tactical  infantry  movements,  the  head  remains  in  a  fixed  alignment  with  the  upper  body  and 
the  upper  body  is  turned  independently  of  the  direction  of  movement. 

4.1.1  Terminology 

We  adopt  terms  used  in  both  vehicular  and  human  navigation  literature  [1,  8]  to  describe  real  world 
locomotion  for  tactical  movement.  There  are  three  components  of  motion  in  the  horizontal  plane.  The  head  is 
assumed  to  be  aligned  with  the  upper-body  as  pictured  in  the  figures. 

•  Heading:  Angular  direction  the  upper-body  faces. 

•  Course:  Angular  direction  the  pelvis  translates. 

•  Displacement:  Distance  the  pelvis  translates. 

The  first  two  components  specify  orientation  with  respect  to  a  fixed  reference  direction  and  the  third  specifies 
distance  travelled.  The  alignment-offset  is  the  angle  between  the  course  and  heading.  The  angle  between  the 
direction  of  a  step  (course)  and  the  direction  of  the  upper  body  (heading)  determines  the  kind  of  step  being 
taken  (Figure  2). 


I  i> 

$  «&>  % 

Course  Heading 

Forward  Side  Backward  Diagonal 

Step  Step  Left  Step  Step 

Figure  2:  Illustration  of  Basic  Terms:  Here  the  direction  of  aim  indicates  the  heading.  The  angle 
between  the  course  and  heading  determines  the  kind  of  step  being  taken. 


4.1.2  Classification  of  Human  Locomotion 

The  following  classification  illustrates  the  kinds  of  motion  a  person  can  make  while  walking  or  running  in  the 
real  world  in  terms  of  how  course  and  heading  vary.  In  the  figures,  all  kinds  of  motion  involve  moving  along 
a  path.  A  person  can  also  turn  his  or  her  heading  without  translating  by  twisting  the  pelvis  or  stepping  to  turn 
in  place,  but  that  is  not  depicted. 

Steering  Motion:  Course  and  heading  remain  co-aligned  while  moving  along  a  path  (Figure  3).  Steering 
motion  can  be  used  to  move  directly  towards  a  stationary  or  moving  target  and  to  follow  a  route  marked  out 
on  the  ground  by  facing  forward  along  the  trail. 
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Figure  3:  Examples  of  Steering,  Canted,  Oblique,  and  Scanning  Motion. 


Canted  Motion:  Course  and  heading  maintain  a  fixed  alignment-offset  while  moving  along  a  path  (Figure  3). 
Steering  motion  is  a  sub-class  of  canted  motion. 

Oblique  Motion:  Heading  remains  in  a  fixed  direction  while  moving  along  a  path  (Figure  3).  Notice  that 
moving  along  a  straight  path  with  a  fixed  alignment-offset  qualifies  as  both  canted  and  oblique  motion,  so  the 
classes  partially  overlap. 

Scanning  Motion:  Heading  is  free  to  turn  separately  from  the  course  while  moving  along  a  path  (Figure  3). 
Scanning  motion  can  be  used  to  search  from  side-to-side  while  moving  or  to  remain  directed  towards  a 
stationary  or  moving  target,  while  traversing  a  straight  or  curved  path.  Scanning  while  traversing  a  curved 
path  is  the  only  case  where  the  heading  and  course  vary  independently. 

4.2  Conventional  Gamepad  Controls 

Gamepads  are  widely  used  to  control  virtual  locomotion  in  many  first-person  shooter  games,  especially  on 
game  consoles.  Modem  gamepads  provide  a  pair  of  thumb-joysticks  used  to  change  the  course  and  heading 
of  the  user’s  avatar.  The  controls  are  applied  in  a  fairly  standard  manner. 

The  right  joystick  turns  the  course  and  heading  together  at  a  rate  set  by  the  amount  of  lateral  deflection. 
(Deflecting  this  joystick  forward  and  backward  pitches  the  view  up  and  down,  but  is  not  part  of  locomotion.) 
We  call  the  right  joystick  the  steering  joystick  (Figure  4,  right).  Pushing  the  steering  joystick  to  its  full 
leftmost  extent  turns  course  and  heading  together  to  the  left  at  a  maximum  angular  rate  (degrees  per  second). 
Pushing  it  only  partially  over  to  the  right  turns  them  to  the  right  at  a  slower  rate. 
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Straight  Ahead  Right 


Offset  Steer 

Direction  Course  Turns  Rate  of  Turn 

from  Heading 

Figure  4:  Conventional  Joystick  Controls.  Right:  Rate  control  over  steering. 

Left:  Directional  control  over  course-offset. 

The  left  joystick  sets  the  rate  of  displacement  by  the  amount  of  deflection  of  the  joystick  and  the  direction  of 
the  course  relative  to  the  current  heading  by  the  angular  direction  in  which  it  is  pushed.  We  refer  to  the  left 
joystick  as  the  offset  joystick  because  it  controls  the  alignment-offset  between  course  and  heading  (Figure  4, 
left).  When  the  offset  joystick  is  engaged,  the  course  is  redirected  and  the  heading  remains  unchanged. 
Figure  2  above  showing  examples  of  alignment-offsets  in  terms  of  stepping  patterns  illustrates  the  patterns: 
Pushing  the  offset  joystick  forward  directs  the  course  forward  (forward  step);  pushing  the  joystick  left  or  right 
moves  the  course  to  the  side  relative  to  the  heading  (side  step);  pushing  the  joystick  back  directs  the  course 
back  (back  step);  and  pushing  the  joystick  on  the  diagonal  produces  a  diagonal  course  relative  to  the  heading 
(diagonal  step).  Pushing  only  the  offset  joystick  without  engaging  the  steering  joystick  moves  the  avatar  in  a 
straight  trajectory  in  the  specified  direction;  deflecting  the  steering  joystick  without  engaging  the  offset 
joystick  turns  the  avatar  in  place. 

In  summary,  the  offset  joystick  redirects  the  course  without  altering  the  heading.  The  steering  joystick 
controls  the  rate  at  which  the  course  and  heading  turn  together.  Therefore,  we  categorize  conventional 
gamepad  controls  as  a  course-offset,  rate-based  steering  control. 


4.2.1  Executing  Different  Classes  of  Motion 

The  steering  and  offset  joysticks  are  manipulated  (control  action)  to  move  the  user’s  avatar  through  the  virtual 
world  (control  effect).  Because  of  the  way  the  joystick  actions  control  course,  some  classes  of  motion  are 
easier  to  achieve  than  others.  Moreover,  redundant  control  over  course  gives  rise  to  a  multitude  of  different 
ways  of  entering  the  same  final  course. 

Steering  motion  requires  deflecting  the  steering  joystick  while  pushing  the  offset  joystick  forward.  The  view 
changes  as  the  heading  turns  and  the  avatar  advances  along  the  trail.  It  is  a  closed  loop  control  because  the 
user  relies  on  visual  feedback  to  determine  the  extent  of  rotation  and  translation.  Both  steering  and 
displacement  are  rate  controlled. 

Oblique  motion  is  accomplished  using  only  the  offset  joystick.  Heading  remains  fixed  in  the  current  direction 
because  no  steering  occurs.  The  user  enters  the  course  simply  by  pushing  the  offset  joystick  in  the  desired 
direction  without  need  of  visual  feedback.  Therefore,  oblique  motion  is  controlled  in  an  open  loop  manner. 
The  user  continually  feels  the  current  direction  of  the  course  offset. 
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Scanning  while  following  a  straight  path  is  difficult  using  conventional  gamepad  controls  (Figure  5).  The 
steering  joystick  must  turn  the  heading  to  scan,  but  the  course  turns  along  with  it  (Figure  5,  left).  To  follow  a 
straight  path  (Figure  5,  right),  the  offset  joystick  must  counterturn  to  cancel  out  this  rotation  of  the  course 
(Figure  5,  middle).  It  is  difficult  to  balance  the  two  forms  of  turning  since  one  joystick  is  controlled  by  rate 
and  the  other  by  direction.  Without  knowing  how  far  the  steering  joystick  changes  the  direction,  the  user  does 
not  know  where  to  point  the  offset  joystick  to  counter  it.  Scanning  while  following  a  curved  path  is  even 
more  difficult  using  such  controls. 


Figure  5:  Course-Offset  Control  -  Turning  only  the  Steering  Joystick  Executes  Steering  Motion. 
Activating  only  of  the  course-offset  joystick  executes  oblique  motion.  Combining  these  particular 
actions  executes  scanning  while  traversing  a  straight  path.  Note  the  complex  interaction  between 
the  two  components  of  motion  required  to  maintain  a  straight  course. 


Conventional  gamepad  controls  favor  visually  directed  motion  because  rate-based  steering  relies  heavily  on 
visual  feedback.  Both  canted  and  scanning  motions  allow  the  avatar  to  turn  to  face  a  target  while  moving 
along  a  path.  Users  compensate  for  the  weakness  in  executing  scanning  motion  by  relying  heavily  on  canted 
motion.  They  adopt  a  control  strategy  called  targeted  canted  motion  in  which  the  heading  is  constantly  turned 
toward  the  target  while  the  alignment-offset  of  the  course  and  heading  remains  fixed  in  order  to  steer  them 
together  as  a  unit.  Targeted  canted  motion  is  performed  by  deflecting  the  steering  joystick  while  holding  the 
offset  joystick  in  a  fixed  direction.  A  family  of  spiral  trajectories  emerges  by  doing  so  (Figure  6).  A 
sophisticated  user  can  smoothly  transition  between  these  trajectories  to  achieve  even  more  complicated  paths. 
The  term  strafing  is  used  in  the  gaming  community  to  denote  oblique  motion,  shooting  while  moving 
obliquely,  and  the  form  of  targeted  canted  motion  just  described.  Circle-strafing  is  the  tactic  of  circling  a 
target  while  facing  toward  it. 
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4.3  Pointman:  a  new  directional  control 

Pointman  uses  a  conventional  dual  joystick  gamepad  in  a  new  way.  The  joystick’s  control  over  virtual 
movement  is  modified  to  direct  course  and  heading  in  a  more  factored  way  that  favors  scanning.  View 
control,  including  pitch,  is  offloaded  to  orientation  sensors. 

With  the  new  approach,  the  user  pushes  the  steering  joystick  (Figure  7,  right)  in  the  angular  direction  of  the 
desired  path  of  motion.  This  provides  directional  control  over  turning  the  heading  and  course  together,  as 
opposed  to  the  conventional  rate  control.  The  offset  joystick  (Figure  7,  left)  now  directs  how  far  the  heading 
turns  away  from  the  course.  We  call  this  a  heading-offset  control.  When  only  the  heading-offset  joystick  is 
operated,  the  heading  turns  to  follow  the  direction  in  which  the  joystick  is  pushed.  When  there  is  no 
displacement,  the  heading  rotates,  turning  the  avatar  in  place. 


Straight  Ahead  Right 


Offset  Steer 

Direction  Heading  Turns 
from  Course 


Straight  Ahead 

0°  Right 


Figure  7:  Heading -Offset  Pointman  Controls.  Right:  directional  control  over  steering. 
Left:  directional  control  over  heading-offset. 


A  technique  called  panning  is  applied  to  limit  the  rate  at  which  the  heading  rotates  under  directional  control. 
In  its  raw  form,  an  abrupt  turn  would  instantaneously  jump  the  heading  to  a  new  direction  and  a  smooth  turn 
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would  rotate  the  heading  at  the  rate  by  which  the  joystick  turns.  Panning  is  set  to  a  maximum  rate 
commensurate  with  how  fast  a  human  can  actually  turn  his  or  her  body. 

We  call  this  a  fully  directional  control,  because  both  the  steering  and  offset  joysticks  express  course  and 
heading  as  the  angular  direction  in  which  the  joysticks  are  pointed.  Directional  control  over  both  joysticks 
allows  users  to  better  coordinate  the  two  because  they  are  now  directed  in  a  similar  way.  Moreover,  users 
continually  feel  the  alignment  of  the  two  orientations  and  can  move  their  avatar  in  an  open  loop  manner.  This 
is  an  ongoing  sensation,  just  as  with  actual  locomotion  where  people  feel  the  alignment  between  their  leg 
motion  and  the  direction  of  their  upper  body. 

Before  either  joystick  is  engaged,  both  the  course  and  heading  initially  correspond  to  the  direction  in  which 
the  avatar's  upper  body  faces.  Once  engaged,  the  angle  of  the  steering  joystick  indicates  the  direction  of  the 
course  relative  to  this  initial  direction.  The  angle  of  the  heading-offset  joystick  indicates  the  heading  relative 
to  the  course.  The  initial  direction  provides  a  fixed  reference  frame  until  the  joysticks  are  released.  The 
avatar's  final  heading  becomes  the  initial  direction  for  the  next  round  of  movements. 

We  categorize  Pointman  as  a  heading-offset,  fully  directional  control.  With  this  approach,  both  joysticks  can 
turn  the  heading,  but  the  steering  joystick  also  directs  the  course.  Steering  is  directed  with  respect  to  the 
environment  in  the  same  way  regardless  of  how  the  heading  is  varied  by  the  offset  joystick.  This  provides 
consistency  of  use. 

We  recommend  offloading  control  over  displacement  to  sliding  foot  pedals  that  mimic  a  person’s  foot  motion 
during  locomotion  and  preserves  the  open  loop  nature  of  walking.  The  two  pedals  are  mechanically  coupled 
so  that  one  goes  forward  as  the  other  goes  back.  The  absolute  displacement  of  the  coupled  pedals  is  scaled 
and  applied  to  displace  the  avatar  along  the  course.  Scaling  is  done  on  a  moment-by-moment  basis  and  can  be 
started  or  stopped  at  any  instant. 

The  preferred  method  for  view  control  is  to  attach  a  three  degree-of-freedom  inertial  sensor  to  an  HMD  that 
reports  all  three  viewing  angles  (yaw,  pitch,  and  roll).  In  this  case,  the  view  rotates  naturally  as  the  head  is 
turned.  Alternatively,  the  tracker  can  be  attached  to  a  gamepad.  Users  can  rotate  their  view  by  tilting  the 
gamepad  with  their  hands.  This  approach  works  well  with  fixed  screen  displays,  ranging  from  desktop 
monitors  to  surround  screen  projection  systems.  A  low-end  approach  is  to  allocate  a  pair  of  (shoulder)  buttons 
on  a  gamepad  to  provide  rate  control  over  raising  and  lowering  the  pitch  of  the  view,  but  that  is  more  difficult 
to  use. 

4.3.1  Executing  Different  Classes  of  Motion 

When  individually  applied,  the  steering  joystick  directs  steering  motion  (Figure  8,  left)  and  the  heading-offset 
joystick  directs  scanning  while  traversing  a  straight  path  (Figure  8,  middle).  Other  motions  are  achieved  by 
combining  the  steering  and  heading-offset  joysticks  in  different  ways.  Scanning  while  following  a  straight 
path  is  easy  using  Pointman  (Figure  8)  because  directing  heading  does  not  redirect  the  course.  The  two 
components  of  motion  are  independent  and  combine  in  a  straightforward  manner. 
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V/Offset  Steei\y 

^/Offset  Steer\y 

Figure  8:  Heading-Offset  Control  -  Activating  only  the  of  steering  joystick  executes  trailing  motion. 

Activating  only  the  offset  joystick  executes  scanning  moving  straight  ahead.  Combining  these 
particular  actions  executes  scanning  while  traversing  a  straight  path.  Foot  pedals  apply  a  constant 
rate  of  displacement  in  all  cases.  Note  how  the  two  components  of  motion  make  independent 

contributions  to  the  resulting  motion. 

Moving  obliquely  requires  extra  work  to  maintain  a  forward  heading.  The  steering  joystick  is  turned  to  go  in 
the  desired  direction  while  the  offset  joystick  is  counterturned  to  maintain  the  heading.  Countertums  can  be 
executed  smoothly  by  revolving  both  joysticks  by  an  equal  amount  in  opposite  directions  (a  revolving 
countertum),  or  abruptly  by  pushing  them  in  opposite  directions,  symmetrically  about  the  median  plane  of  the 
gamepad  (a  direct  angle  countertum).  This  can  be  done  without  visual  feedback  and  maintains  the  open  loop 
nature  of  Pointman.  On  balance,  oblique  motion  is  easier  to  execute  with  the  course-offset  control,  and 
scanning  motion  is  easier  to  execute  with  the  heading-offset  control. 

4.4  Comparing  Pointman  to  the  Conventional  Gamepad 

A  review  of  the  literature  on  infantry  tactics  shows  that  scanning,  looking  over  top  of  the  rifle  for  threats  while 
moving,  is  a  defining  element  of  tactical  movement.  But  scanning  is  difficult  with  the  vast  majority  of  device- 
based  virtual  locomotion  controls  which  are  course-offset,  such  as  the  conventional  gamepad  controls,  mouse- 
keyboard  interfaces,  and  interfaces  in  which  a  joystick  is  mounted  on  a  rifle  prop  to  control  the  course  while 
the  user  turns  wearing  an  HMD.  In  all  these  interfaces,  turning  the  heading  redirects  the  course,  dismpting  the 
user’s  ability  to  follow  a  path  while  scanning.  As  a  result,  it  becomes  very  difficult  to  execute  realistic  tactical 
maneuvers. 

In  contrast,  Pointman  adopts  a  heading-offset  control  to  allow  the  user  to  easily  scan  to  look  side-to-side  or 
fixate  on  a  target  while  traversing  a  path  in  the  environment,  corresponding  more  closely  to  recommended 
tactical  movements.  The  steering  joystick  is  manipulated  in  the  same  way  to  traverse  a  given  path  no  matter 
how  the  offset  joystick  alters  heading.  This  independence  offers  additional  advantages.  The  constancy  makes 
it  easier  to  follow  paths  regardless  of  how  users  direct  their  heading,  and  it  reinforces  the  open  loop  approach 
because  fewer  control  variables  need  to  be  sensed  and  acted  on.  With  practice,  users  can  execute  a  variety  of 
real  world  tactical  infantry  movements  in  a  simulator  using  either  an  HMD  or  fixed  display. 
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Solving  the  tactical  movement  problem  led  to  a  control  that  allows  users  to  freely  scan  their  heading  without 
redirecting  the  course.  This  result  can  benefit  any  application  where  it  is  useful  to  freely  look  around  while 
continuing  to  move. 


CONCLUSION 

In  summary,  we  have  described  two  interfaces  for  virtual  infantry  training  simulators.  Gaiter  is  a  high-end 
body-driven  interface  that  allows  the  user  to  apply  and  intermix  a  range  of  natural  and  gestural  actions. 
Pointman  is  a  new  device-driven  interface  that  allows  the  user  to  independently  direct  the  course  and  heading 
to  move  about  in  the  virtual  world.  Both  allow  the  user  to  execute  the  recommended  tactical  infantry 
movements.  The  two  interfaces  can  be  applied  by  different  users  participating  in  a  common  networked 
simulation.  Each  trainee  has  an  avatar  that  reflects  his  or  her  actions,  allowing  team  members  to  coordinate 
their  actions.  The  advantage  of  Gaiter  is  that  it  engages  the  user's  entire  body  in  the  simulation  and  requires 
correct  manipulation  of  a  realistic  rifle  prop.  Pointman  uses  the  gamepad  for  turning  and  moving.  Realistic 
weapons  handling  is  traded-off  to  provide  a  low-cost,  small-footprint  system  that  can  easily  be  set-up  and 
operated  anywhere. 


5.0  ACKNOWLEDGEMENTS 

This  work  was  sponsored  by  the  Naval  Research  Laboratory  internal  funding  and  the  Office  of  Naval 
Research  (ONR).  The  work  is  part  of  the  Virtual  Technologies  and  Environments  (VIRTE)  program  under 
the  direction  of  CDR  Dylan  Schmorrow,  which  is  part  of  the  Capable  Manpower  Future  Naval  Capabilities 
(FNC)  Program  at  ONR. 


6.0  REFERENCES 

[1]  Beall,  A.C.  and  Loomis,  J.M.  (1996).  Visual  control  of  steering  without  course  information. 
Perception ,  25,  481-494. 

[2]  Chance,  S.S.,  Gaunet ,  F.,  Beall,  A.C.,  and  Loomis,  J.M.  (1997).  Locomotion  mode  affects  the  updating 
of  objects  encountered  during  travel:  The  contribution  of  vestibular  and  proprioceptive  inputs  to  path 
integration.  Presence  7(2),  pp.  168-178. 

[3]  Fukusima,  S.S.,  Loomis,  J.M.,  and  Da  Silva,  J.A.  1997.  Visual  perception  of  egocentric  distance  as 
assessed  by  triangulation.  Journal  of  Experimental  Psychology:  Human  Perception  and  Performance , 
23(1).  86-100. 

[4]  Gibson,  J.J.  (1986).  The  ecological  approach  to  visual  perception.  Hillsdale,  NJ:  Lawrence  Erlbaum 
Associates,  Inc. 

[5]  Kennedy,  R.S.,  Lane,  N.E.,  Berbaum,  K.S.,  and  Lilienthal,  M.G.,  1993.  Simulator  sickness 
questionnaire:  An  enhanced  method  for  quantifying  simulator  sickness.  The  International  Journal  of 
Aviation  Psychology  3(3),  203-220. 

[6]  Klatzky,  R.L.,  Loomis,  J.M.,  Beall,  A.C.,  Chance,  S.S.,  and  Golledge,  R.G.  (1998).  Spatial  updating  of 
self-position  and  orientation  during  deal,  imagined,  and  virtual  locomotion.  Psychological  Science ,  9(4), 
293-298. 


RTO-MP-HFM-1 36 


23-15 


Immersive  Simulation  to  Train  Urban  Infantry  Combat 


ORGANIZATION 


[7]  Loomis,  J.M.,  Da  Silva,  J.A.,  Fujita,  N.,  &  Fukusima,  S.S.  (1992).  Visual  space  perception  and  visually 
directed  action.  Journal  of  Experimental  Psychology:  Human  Perception  and  Performance ,  18(4),  906- 
921. 

[8]  Loomis,  J.M.,  Klatzky,  R.L.,  Golledge,  R.G.,  and  Philbeck,  J.W.  (1999).  Human  navigation  by  path 
integration.  In  R.G.  Golledge  (Ed.),  Wayfinding  Behavior:  Cognitive  mapping  and  Other  Spatial 
Processes  (pp.  125-151).  Baltimore:  Johns  Hopkins  University  Press. 

[9]  Marine  Corps  Institute.  (1997).  Military  operations  on  urban  terrain.  (Marine  Corps  Institute  Report 
MCI  03.66b).  Washington,  DC:  Marine  Barracks. 

[10]  Philbeck,  J.W.  &  Loomis,  J.N.M.  (1997).  Comparison  of  two  indicators  of  perceived  egocentric 
distance  under  full-cue  and  reduced-cue  conditions.  Journal  of  Experimental  Psychology:  Human 
Perception  and  Performance ,  25(1),  72-85. 

[11]  Philbeck,  J.W.,  Loomis,  J.M.,  &  Beall,  A.C.  (1997).  Visually  perceived  location  is  an  invariant  in  the 
control  of  action.  Perception  &  Psychophysics ,  59(4),  601-612. 

[12]  Reiser,  J.J.,  Ashmead,  D.H.,  Talor,  C.R.,  &  Youngquist,  G.A.  (1990).  Visual  perception  and  the 
guidance  of  locomotion  without  vision  to  previously  seen  targets.  Perception ,  19,  675-689. 

[13]  Slater,  M.,  Usoh,  M.,  and  Steed,  A.  (1995).  Taking  steps:  The  influence  of  a  walking  technique  on 
presence  in  virtual  reality.  ACM  Transactions  on  Computer-Human  Interaction ,  2(3),  pp.  201-219. 

[14]  Sibert,  L.E.,  Templeman,  J.N.,  Page,  R.C.,  Barron,  J.T.,  McCune,  J.A.,  and  Denbrook,  P.S.  (2004). 
Initial  assessment  of  human  performance  using  the  gaiter  interaction  technique  to  control  locomotion  in 
fully  immersive  virtual  environments.  (NRL/FR/5510 — 04-10,086).  Washington,  DC:  Naval  Research 
Laboratory. 

[15]  Suarez,  G.  (1998).  The  tactical  advantage.  Boulder,  CO:  Paladin  Press. 

[16]  Templeman,  J.N.,  Denbrook,  P.S.,  and  Sibert,  L.E.  (1999).  Virtual  locomotion:  Walking  in  place 
through  virtual  environments.  Presence ,  8(6),  pp.  598-617. 

[17]  Templeman,  J.N.  and  Sibert,  L.E.  (in  press).  Immersive  simulation  of  coordinated  motion  in  virtual 
environments.  In  G.  Allen  (Ed.),  Applied  Spatial  Cognition:  From  Research  to  Cognitive  Technology. 
Lawrence  Erlbaum  Associates,  Inc. 


23-16 


RTO-MP-HFM-136 


NRL 


HUMAN  FACTORS  &  MEDICINE 
PANEL  WORKSHOP 


Immersive  Simulation  to  Train 
Urban  Infantry  Combat 

Jim  Templeman 

(jim.templeman@nrl.navy.mil) 

Naval  Research  Laboratory 
Code  551 1 :  Immersive  Simulation 

Virtual  Media  for  Military  Applications 
(Unclassified) 

13- 15  June  2006 

O  RGANIZATTON 

U.S.  Military  Academy  West  Point 


The  Need  for  Realistic  Tactical 
Virtual  Simulators  for  the  Infantry 


NRL 


•  Accurately  represented  dismounted  infantry  remains  a  key 
element  missing  from  today’s  virtual  simulations 

-  For  training,  mission  preparation,  &  virtual  prototyping 

of  infantry  &  combined  arms  (tactics,  equipment,  and  scenarios) 

•  Control  over  a  virtual  human  is  not  well  understood 

-  Humans  can  look,  move,  &  shoot  with  a  high  level  of 
coordination  &  maneuverability 

-  Virtual  locomotion:  allows  a  user  to  move  through  the 
virtual  world,  while  remaining  in  a  small  physical  space 


Full-Body  Immersive  Avatar 

driven  by  the  user’s  actions 


NRL 


•  Avatar:  depiction  of  a  person's  body  in  virtual  reality 

-  Track  and  render  user’s  full-body  and  rifle 

-  Teammates  see  each  other’s  Avatar  move  in  real  time 

-  Avatar  provides  primary  means  of  manipulating  virtual  objects 

•  Gaiter:  virtual  locomotion  control 

-  User  walks  in  place  to  move  through  VE 


View  of  User’s  Avatar  Physical  Set-Up 

-  Real  Time  Interaction  -  Cameras  track  markers 

-  Look  -  Move  -  Shoot  -  Head-Mounted  Display  3 
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Tactical  Infantry  Movement 


•  Tactical  movement  requires  covering  potential  danger 
areas  as  they  are  encountered 


Alert  Carry:  a  Marine  scans  for  threats  with  the  rifle  aligned  below 
the  line  of  sight  -  “guns  ’n’  eyeballs ” 

-  The  rifle  is  snapped  into  a  firing 
position  when  a  threat  is  spotted 

-  Once  the  threat  is  eliminated,  the 
Marine  returns  to  the  alert  carry 
and  continues  to  scan  for  threats 


•  The  upper  body  turns  as  a  unit 


-  As  a  “Turret” 


Heading 


f 

Course 


Alert  Carry  Ready  Carry 


Forward  Side  Step  Backward  Diagonal 
Step  Left  Step  Step 
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Simulating  Tactical  Infantry  Movement 


•  Motion  strategies  adopted  in  first-person-shooter  games 
do  not  match  actual  infantry  tactics 

-  Typical  gamepads  &  mouse-keyboard  controls  emphasize  strafing 

-  Actual  tactical  movements  rely  heavily  on  scanning  while  moving  to 
cover  all  immediate  danger  areas 

\  Forward  Security 
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Analysis: 

Different  Classes  of  Motion 
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Conventional  Controls 

Rate-Based  Steering  &  Oblique  Control 


•  Conventional  Gamepad  Controls  for  First-Person-Shooter  Games 


Directional  Control  over 
Oblique  Motion 


Rate  Control  over  Steering 


Conventional  Controls 

Favors  Oblique  Motion 
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-  Heading  can  only  be  directed  by  steering 

•  But  steering  redirects  the  course! 

-  Course  must  then  be  readjusted  by  the  oblique  control 

•  Difficult  to  coordinate  rate-based  &  directional  control 


Mouse/keyboards  &  HMD/rifle-joysticks  also  apply  oblique  control 
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Pointman 

Directional  Steering  &  Scan  Control 
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•  Modify  the  operation  of  both  joysticks 


Directional  Control  Directional  Control  over  Steering 

over  Scanning  Motion  -  Makes  coordination  easier 

-  User  continually  senses  direction 


Pointman 

Favors  Scanning  Motion 
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-  Ease  in  executing  scanning  motion 

•  Scanning  directs  heading  without  redirecting  the  course 

-  Steering  is  always  directed  in  the  same  way  to  follow  the  same  path 

•  No  matter  how  the  heading  is  varied 
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Pointman 

Low  Cost,  Small  Footprint,  Ship  Deployable 
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•  Scanning  while  moving  along  a 

curved  path  is  the  true  test  of  a 
control’s  ability  - 

•  Add  orientation 
sensors  to  gamepad 
or  a  head-mounted 
display,  for  view 
control 

Add  sliding  foot- 
pedals  to  allow 
measured  steps  to  be 
taken 

•  Freedom  to  scan  looking  over  a 
gunsight  translates  into  a  general 
freedom  to  look  around  in  less 
specialized  applications 


Oblique  Control 
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Differences  Between  Practicing 

First-Person-Shooter  &  Real  Infantry  Tactics 
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Novices  apply 
steering  motion 
until  they  gain  skill 
at  coordinating 
course  &  heading 


Trainees  get  more 
chance  to  practice 
using  low-cost 
accessible  systems 


Steering  Motion 
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Tactical  Movement  with  Pointman 
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Urban  combat  is  three-dimensional  warfare 
because  you  can  be  shot  from  all  around 


Pointman 


Pointman  in  Action 
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Advantages  of  Simulations  Using 
Virtual  Locomotion 


•  Accessible 

-  Practice  anytime,  anywhere 

-  Model  terrain  you  do  not  have  physical  access  to 

-  Remote  sites  can  share  a  common  virtual  space 

•  Safe 

-  Practice  high  risk  conditions 

•  Explosive  breaching 

•  Even  stairwells  with  accumulated  brass  can  be  hazardous 

•  Inexpensive 

-  Minimal  energy,  travel,  &  pollution  issues 

•  Fully  Logged 

-  Accurate  logging  of  every  action  performed 

•  Simulation  complements  rather  than  replaces  RW  methods 
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AMD  EHVI  ROMMEbJTS 


Path  Integration  Experiment 


•  Path  Integration:  kinesthetic  &  vestibular  inputs  contribute  to  a 
person’s  sense  of  position  in  the  real  world 

-  People  are  good  at  judging  distance  traveled  and  angle  turned  when 
walking  blindfolded 

•  Hypothesis:  kinesthetic  &  vestibular  input  help  with  path 
integration  in  the  virtual  world,  especially  without  vision 

-  Performed  in  collaboration  with  Warfighter  HSI  Lab  (WHSIL)  at  NRL 


Gaiter 


Rifle-Mounted 

Joystick 


Desktop  Joystick 

with  Non-T racked 
Head-Mounted  Display 
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Comparison  of  Locomotion  Controls 

(between  subjects,  8  subjects  per  group) 


Straight  Ahead 


90° 
Right 

180° 


Maze  Task:  significant  difference  among  controls 

•  (F(2,  90)  =  4.47,  p  =  .024) 

-  Desktop  Joystick  produced  less  angular  error  than  the 
Rifle-Mounted  Joystick 

Blind  Turning:  significant  difference 

•  (F(2,  146)  =  5.72,  p  =  .01) 

-  Gaiter  &  Rifle-Mounted  Joystick  were  more  accurate 
than  Desktop  Joystick 

Blind  Walking:  significant  difference 

•  (F(2,  100)  =  3.53,  p  =  .047) 

-  Rifle-Mounted  Joystick  was  more 
accurate  than  Gaiter 

-  Scaling  distances  traveled  using  Gaiter 
by  2X  eliminated  the  difference 

•  (F(2,  99)  =  0.833,  p  =  .44) 
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Virtual  Marksmanship 


Does  extensive  practice  using  a  Rifle  Prop  &  H 

degrade  Shooting  Skills? 


Rapid  Engagement  of  Targets 

-  Visually  Acquire  Targets:  focus;  eye,  head,  &  body  motions 

-  Rapid  Indexing:  developing  a  ‘reflexive’  ability  to  bring  weapon  up 
with  the  sights  nearly  aligned  on  target  (0.5°) 

-  Timing:  just  long  enough  to  adjust  to  hit  the  target 


•  Points  of  Failure 

-  Physical  Misalignment:  incorrect  tracking 

-  Optical  Misalignment:  misplacement  of  image  on  the  retina 

-  Latency:  tracking  +  simulation  +  rendering  +  display 

-  Image  Artifacts:  rendering  +  display  +  optics 

-  Reduced  Field-of-View:  false  indexing  (medial  sight  placement) 

-  Depth  of  Focus:  unavailable 

-  Operation  of  Weapon  Prop:  missing  functions? 

_  NanH  £.  Pinnor  TrorUnn  t fnr  nriolfi  inrfinn  ri  ri  1 1  c\  ■  nmhlQmcitir 
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